Image processing system, image processing method, and program

ABSTRACT

The present disclosure relates to an image processing system, an image processing method, and a program capable of achieving better communication. An information processing apparatus includes a clipping unit configured to perform image processing of clipping a predetermined object from a moving image in which the object is imaged and generate a clipped moving image of the object, and an operation content acquisition unit configured to obtain operation content of operation performed on the clipped moving image of the object. A distribution server includes an image processing unit configured to perform, individually for each of the objects, image processing that corresponds to operation content on a clipped moving image of a plurality of objects, and a combining unit configured to generate a combined moving image in which the plurality of objects is arranged by combining the clipped moving images of the plurality of objects that has undergone image processing. The present technology is applicable, for example, to a communication system that enables communication among a plurality of users at remote sites using an image.

TECHNICAL FIELD

The present disclosure relates to an image processing system, an imageprocessing method, and a program, and particularly relates to an imageprocessing system, an image processing method, and a program capable ofachieving better communication.

BACKGROUND ART

In recent years, services that provide communication systems usingmoving images, such as video telephony, video conference, and open livebroadcast by a user have been increasing. For example, the communicationsystem includes a use case in which a plurality of users present atremote sites performs joint appearance on a same screen.

As a technique of presenting such joint appearance, there is a mainlyused technique of arranging moving images captured by video cameras heldat individual remote sites to be positioned adjacent to each other onthe same screen. While this technique enables simultaneous viewing ofstates of the individual users, it is difficult to obtain a realisticfeeling or togetherness to indicate that the users are performingdistribution by joint appearance, since the screen includes merelyindividual moving images of the plurality of users positioned adjacentto each other.

In contrast, Patent Document 1 proposes a technology in which the userclips a portion of the user oneself from the captured moving image usinga parallax camera and arranges the portion in a same space shared by ajoint user. Moreover, Patent Document 2 proposes a technology in which auser operates the size and movement of a virtual object including theuser oneself by using gesture on a television device.

CITATION LIST Patent Document

-   Patent Document 1: Japanese Patent Application Laid-Open No.    2014-238731-   Patent Document 2: Japanese Patent Application Laid-Open No.    2000-197030

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

Meanwhile, with the technology proposed in Patent Document 1 describedabove, it is merely possible to display a clipped user portion at apredetermined position such as “next to the other user” and it isdifficult for the user to operate freely in determining the method ofdisplay. Moreover, according to the technology proposed in PatentDocument 2 described above, for example, it is difficult to performcorresponding operation in a case where a clipped region of the useroneself is excessively larger than the user-intended size, making itdifficult to appropriately clip the portion of the user oneself.

In this manner, the conventional technology may have difficulty inappropriately reflecting user's operation in the communication system,leading to a difficulty in achieving better communication.

The present disclosure has been made in view of this situation, and apurpose of the present disclosure is to achieve better communication.

Solutions to Problems

An image processing system according to an aspect of the presentdisclosure includes an image processing unit configured to perform,individually for each of objects, image processing on an object movingimage generated from a moving image in which a predetermined object isimaged and generated so as to include a region in which the object isphotographed, the image processing corresponding to operation contentapplied to the object moving image, and a combined image generationprocessing unit configured to generate a combined moving image includingat least one of the objects by combining the object moving image thathas undergone image processing with another moving image.

An image processing method or a program according to an aspect of thepresent disclosure includes the steps of performing, individually foreach of objects, image processing on an object moving image generatedfrom a moving image in which a predetermined object is imaged andgenerated so as to include a region in which the object is photographed,the image processing corresponding to operation content applied to theobject moving image, and generating a combined moving image including atleast one of the objects by combining the object moving image that hasundergone image processing with another moving image.

According to an aspect of the present disclosure, image processing isperformed individually for each of objects, the image processingcorresponding to operation content applied to an object moving image, onthe object moving image generated from a moving image in which apredetermined object is imaged and generated so as to include a regionin which the object is photographed, and a combined moving image inwhich at least one object is arranged is generated by combining aplurality of the object moving images that has undergone imageprocessing with another moving image.

Effects of the Invention

According to one aspect of the present disclosure, it is possible toachieve better communication.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an exemplary configuration of acommunication system according to an embodiment of the presenttechnology.

FIG. 2 is a view for describing a combined moving image distributed in acommunication system.

FIG. 3 is a block diagram illustrating an exemplary configuration of aninformation processing apparatus.

FIG. 4 is a block diagram illustrating an exemplary configuration of adistribution server.

FIG. 5 is a view for describing operation of free deformation applied toa user-clipped moving image.

FIG. 6 is a view for describing operation of removing an unnecessaryregion applied to a user-clipped moving image.

FIG. 7 is a diagram for describing image processing based on likelihoodinformation.

FIG. 8 is a diagram for describing processing of continuously performingdeletion of an unnecessary region.

FIG. 9 is a view for describing operation of setting a border designaround a user-clipped moving image.

FIG. 10 is a view for describing trimming operation applied to auser-clipped moving image.

FIG. 11 is a view for describing operation of designating depthcoordinates applied to a user-clipped moving image.

FIG. 12 is a view for describing depth coordinates.

FIG. 13 is a diagram illustrating a format of operation content.

FIG. 14 is a flowchart for describing processing performed on theinformation processing apparatus.

FIG. 15 is a flowchart for describing processing performed on thedistribution server.

FIG. 16 is a block diagram illustrating a modification example of theinformation processing apparatus.

FIG. 17 is a view for describing a change in display due to interactionbetween user-clipped moving images.

FIG. 18 is a view for describing processing of reflecting preregisteredoperation content at a designated time.

FIG. 19 is a view for describing a method of determining operationcontent to be reflected.

FIG. 20 is a block diagram illustrating an exemplary configuration of acomputer according to an embodiment of the present technology.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, specific embodiments of the present technology will bedescribed in detail with reference to the drawings.

FIG. 1 is a block diagram illustrating an exemplary configuration of acommunication system according to an embodiment of the presenttechnology.

A communication system 11 in FIG. 1 provides a service for achievingcommunicating among a plurality of users located at remote sites usingan image and enables image processing in the entire system. For example,as illustrated in the figure, the communication system 11 includes aplurality of information processing apparatuses 13 and a distributionserver 14 connected via a network 12 such as the Internet. For example,FIG. 1 illustrates an exemplary configuration in which three informationprocessing apparatuses 13-1 to 13-3 are connected with each other viathe network 12.

Each of the information processing apparatuses 13-1 to 13-3 is, forexample, a camera device with a communication function, and generates auser-clipped moving image by clipping a region in which a user isphotographed from a moving image obtained by imaging each of users.Then, each of the information processing apparatuses 13-1 to 13-3transmits the corresponding generated user-clipped moving image to thedistribution server 14 via the network 12. Note that the detailedconfiguration of the information processing apparatus 13 will bedescribed below with reference to FIG. 3.

The distribution server 14 collects all the user-clipped moving imagestransmitted from the information processing apparatuses 13-1 to 13-3 viathe network 12 and performs image combining processing of superimposingthe collected user-clipped moving image on a predetermined backgroundmoving image. Then, the distribution server 14 distributes a combinedmoving image generated by the image combining processing to theinformation processing apparatuses 13-1 to 13-3 via the network 12. Notethat the detailed configuration of the distribution server 14 will bedescribed below with reference to FIG. 4.

For example, the combined moving image distributed in the communicationsystem 11 in FIG. 1 will be described with reference to FIG. 2.

A left side portion of FIG. 2 illustrates moving images A1 to A3 astargets of image processing on the information processing apparatuses13-1 to 13-3, and the right side of the moving images A1 to A3illustrates user-clipped moving images B1 to B3 obtained by clipping theusers photographed in the moving images A1 to A3. The user-clippedmoving images B1 to B3 are transmitted to the distribution server 14,and as illustrated on the right side portion of FIG. 2, a combinedmoving image C obtained by combining the user-clipped moving images B1to B3 with a predetermined background moving image is generated anddistributed from the distribution server 14.

Note that the information processing apparatuses 13-1 to 13-3 each havea similar configuration. Hereinafter, therefore, in a case where thereis no need to distinguish between the information processing apparatuses13-1 to 13-3, they will be simply referred to as the informationprocessing apparatus 13. Moreover, the moving images A1 to A3 and theuser-clipped moving images B1 to B3 will also be referred to as themoving image A and the user-clipped moving image B in a similar manner.

Next, FIG. 3 is a block diagram illustrating an exemplary configurationof the information processing apparatus 13 in FIG. 1.

As illustrated in FIG. 3, the information processing apparatus 13includes a digital signal processing unit 21, a distance measurementunit 22, a clipped image generation unit 23, a communication unit 24, adisplay unit 25, and an operation unit 26.

A moving image A (refer to FIG. 2) based on RGB signals captured by animaging device (not illustrated) is supplied to the digital signalprocessing unit 21, for example. Then, the digital signal processingunit 21 applies, to the moving image A, digital signal processing neededfor performing image processing in the clipped image generation unit 23,whereby, for example, the digital signal processing unit 21 obtainsimage data for each of frames constituting the moving image A andsupplies the image data to the clipped image generation unit 23.

The distance measurement unit 22 measures a distance to an objectphotographed in the moving image A supplied to the digital signalprocessing unit 21. For example, a plurality of moving images capturedby at least two imaging devices (not illustrated) with a samecomposition as that of the moving image A is supplied to the distancemeasurement unit 22. Then, the distance measurement unit 22 calculates aparallax between these moving images, thereby measuring the distance tothe object photographed in the moving image A. Then, the distancemeasurement unit 22 generates a depth map that maps the distance fromthe imaging device to the object for each of pixels, and supplies thedepth map to the clipped image generation unit 23.

On the basis of the depth map supplied from the distance measurementunit 22, the clipped image generation unit 23 performs, on the movingimage A supplied from the digital signal processing unit 21, imageprocessing of clipping a target object from the moving image A bymasking a target object region in which the target object as a target ofclipping is photographed. For example, in a case where the user isphotographed as illustrated in FIG. 2, the clipped image generation unit23 sets the user as the target object on the basis of the depth map andgenerates a user-clipped moving image B (refer to FIG. 2) in which theuser alone is clipped from the moving image A. Then, the clipped imagegeneration unit 23 supplies the generated user-clipped moving image B tothe communication unit 24.

The communication unit 24 transmits the user-clipped moving image Bsupplied from the clipped image generation unit 23 to the distributionserver 14 via the network 12. Moreover, if the user performs operationusing the operation unit 26, the communication unit 24 adds theoperation content to the user-clipped moving image B and transmits theimage with the operation content. Furthermore, the communication unit 24receives the combined moving image C (refer to FIG. 2) distributed fromthe distribution server 14 and supplies the combined moving image C tothe display unit 25.

The display unit 25 includes, for example, a liquid crystal display oran organic electroluminescence (EL) display, and displays the combinedmoving image C received by the communication unit 24. Moreover, thedisplay unit 25 displays a graphical user interface that displaysvarious types of operation content needed for the user to performoperation on the user-clipped moving image B combined in the combinedmoving image C by using the operation unit 26.

The operation unit 26 includes a touch panel arranged on a surface ofthe display unit 25, for example, obtains operation content input byuser's operation of touching the user interface displayed on the displayunit 25, and supplies the obtained operation content to thecommunication unit 24.

The information processing apparatus 13 configured as described above iscapable of transmitting the user-clipped moving image B to thedistribution server 14, and receiving and displaying the combined movingimage C distributed from the distribution server 14. Moreover, in a casewhere the user performs operation, the information processing apparatus13 is capable of adding the operation content to the user-clipped movingimage B, transmitting the image with the operation content, andreceiving the combined moving image C on which the operation content hasbeen reflected.

Next, FIG. 4 is a block diagram illustrating an exemplary configurationof the distribution server 14 in FIG. 1.

As illustrated in FIG. 4, the distribution server 14 includes acommunication unit 31, a user management unit 32, user individualprocessing units 33-1 to 33-3, a background recording unit 34, and anoverall image combining unit 35.

The communication unit 31 receives the user-clipped moving image Btransmitted from the communication unit 24 of the information processingapparatus 13 via the network 12 and supplies the received user-clippedmoving image B to the user management unit 32. Moreover, thecommunication unit 31 transmits the combined moving image C suppliedfrom the overall image combining unit 35 to the information processingapparatus 13 via the network 12.

The user management unit 32 performs processing of allocating theuser-clipped moving images B for each of the information processingapparatuses 13 that have transmitted the user-clipped moving images Bsupplied from the communication unit 31. For example, in a case wherecommunication is performed by the users of the information processingapparatuses 13-1 to 13-3, as illustrated in FIG. 1, the user managementunit 32 allocates the user-clipped moving images B1 to B3, transmittedfrom the information processing apparatuses 13-1 to 13-3, to the userindividual processing units 33-1 to 33-3, respectively. Specifically,the user management unit 32 supplies the user-clipped moving image B1 tothe information processing apparatus 13-1, supplies the user-clippedmoving image B2 to the information processing apparatus 13-2, andsupplies the user-clipped moving image B3 to the information processingapparatus 13-3.

The user individual processing unit 33-1 includes an individual imagerecording unit 41-1, an operation recording unit 42-1, and an imageprocessing unit 43-1.

The individual image recording unit 41-1 records the user-clipped movingimage B1 of the information processing apparatus 13-1 supplied from theuser management unit 32. In a case where operation content has beenadded to the user-clipped moving image B1 of the information processingapparatus 13-1 supplied from the user management unit 32, the operationrecording unit 42-1 extracts and records the operation content. Theimage processing unit 43-1 performs, on the user-clipped moving image B1of the information processing apparatus 13-1 recorded in the individualimage recording unit 41-1, imaging processing in accordance with theoperation content in the information processing apparatus 13-1 recordedin the operation recording unit 42-1, and supplies the processed imageto the overall image combining unit 35.

At this time, the individual image recording unit 41-1 updates andrecords the user-clipped moving image B1 frame by frame. Meanwhile, theoperation recording unit 42-1 updates and records the operation contentsolely when the operation content has been added to the user-clippedmoving image B1. Accordingly, the image processing unit 43-1continuously performs the image processing based on the same operationcontent on the user-clipped moving image B1 until the operation contentrecorded in the operation recording unit 42-1 is updated.

Similarly to the user individual processing unit 33-1, the userindividual processing units 33-2 and 33-3 include individual imagerecording units 41-2 and 41-3, operation recording units 42-2 and 42-3,and image processing units 43-2 and 42-3, respectively. Similarly to theuser individual processing unit 33-1, the user individual processingunits 33-2 and 33-3 perform image processing on the user-clipped movingimages B2 and B3, respectively, and supply the processed images to theoverall image combining unit 35.

The background recording unit 34 records various background movingimages as backgrounds of the user-clipped moving images B1 to B3 in thecombined moving image C distributed from the distribution server 14 andsupplies predetermined background moving image to the overall imagecombining unit 35.

The overall image combining unit 35 generates the combined moving imageC by performing image processing of combining the user-clipped movingimages B1 to B3 that have undergone image processing individually in theuser individual processing units 33-1 to 33-3 with the background movingimage supplied from the background recording unit 34. Then, the overallimage combining unit 35 supplies the generated combined moving image Cto the communication unit 31, and distributes the combined moving imageC to the information processing apparatuses 13-1 to 13-3 via the network12.

The distribution server 14 configured as described above is capable ofindividually performing image processing reflecting the operationcontent, on the user-clipped moving images B1 to B3. Then, thedistribution server 14 is capable of distributing the combined movingimage C obtained by combining the user-clipped moving images B1 to B3 inwhich the operation content has been reflected, to the informationprocessing apparatuses 13-1 to 13-3.

In this manner, the communication system 11 including the informationprocessing apparatuses 13-1 to 13-3 and the distribution server 14enables users of the information processing apparatuses 13-1 to 13-3 tocommunicate with each other by the combined moving image C arranged in asame virtual space. Accordingly, the communication system 11 enables aplurality of users located at remote sites to communicate with eachother with enhanced realistic feeling or togetherness, compared with acommunication system in which the moving images A1 to A3 are arranged asthey are on the same screen, for example.

Furthermore, the communication system 11 enables each of the users toperform various types of operation on one's user-clipped moving image B,for the combined moving image C displayed on the display unit 25 of theinformation processing apparatus 13. Since individual operation contentis reflected in real time, the user can concentrate on communicationwithout feeling stress such as one felt when intended operation is notreflected. Accordingly, the communication system 11 makes it possible toachieve better communication among a plurality of users at remote sites.

Next, user's operation performed on the combined moving image Cdisplayed on the display unit 25 of the information processing apparatus13 will be described with reference to FIGS. 5 to 12.

FIG. 5 illustrates an example in which the user of the informationprocessing apparatus 13-1 performs operation of free deformation on theuser-clipped moving image B1.

For example, the user of the information processing apparatus 13-1touches and designates the user-clipped moving image B1 displayed on thedisplay unit 25, and thereafter operates a button (not illustrated) fordesignating operation of free deformation. In response to this, asillustrated in FIG. 5, the operation unit 26 displays the combinedmoving image C1 on which the user interface D1 for performing freedeformation is superimposed, on the display unit 25 so as to enclose theuser-clipped moving image B1 in a rectangle.

Then, the user can enlarge the user-clipped moving image B1 by moving agrid displayed at each of four corners or centers of four sides of theuser interface D1 outward while touching the grid and can reduce theuser-clipped moving image B1 by moving the grid inward while touchingthe grid. Moreover, the user can translate the user-clipped moving imageB1 by moving any one point inside the user interface D1 while touchingthe point. Moreover, the user can rotate the user-clipped moving imageB1 by relatively rotating any two points inside the user interface D1while touching the two points.

In accordance with the operation, for example, operation content (forexample, touch position) is added to the user-clipped moving image B1frame by frame and transmitted from the information processing apparatus13-1. Subsequently, in the distribution server 14, the image processingunit 43-1 can perform image processing according to the operationcontent on the user-clipped moving image B1. Accordingly, theinformation processing apparatus 13-1 makes it possible to display thecombined moving image C1 in which the free deformation using the userinterface D1 has been reflected on the user-clipped moving image B1, ina state where the user is moving.

This enables the user of the information processing apparatus 13-1 tochange the size and arrangement of the user-clipped moving image B1 inreal time while confirming the size and arrangement of the user-clippedmoving images B2 and B3 combined in the combined moving image C1.

FIG. 6 illustrates an example in which the user of the informationprocessing apparatus 13-1 performs operation of removing an unnecessaryregion of the user-clipped moving image B1.

For example, when the clipped image generation unit 23 performs imageprocessing of clipping the user-clipped moving image B1 from the movingimage A1, there might be a case where an unnecessary region to bedeleted remains as a result of erroneous recognition of the region inwhich the user is displayed. For example, a combined moving image C2illustrated in FIG. 6 indicates a state where an unnecessary regionremains on the outside of the right arm of the user-clipped moving imageB1, as indicated by the broken line.

In this case, if the user touches the user-clipped moving image B1 byoperating a button (not illustrated) to designate removal of theunnecessary region, an eraser-shaped user interface D2 for removing theunnecessary region is superimposed and displayed on the combined movingimage C2. Then, if the user rubs the unnecessary region while touchingthe region, the unnecessary region turns into a non-display state.Moreover, if the user operates a button (not illustrated) fordesignating cancellation of removal of the unnecessary region, thedeletion of the unnecessary region is reset, and the unnecessary regionin the non-display state is displayed.

At this time, the image processing unit 43 can hold likelihoodinformation indicating the likelihood of being an unnecessary region foreach of region pixels and can perform image processing of removing thepixel such that the higher the likelihood, the pixel is removed to havehigher transparency.

For example, image processing based on likelihood information will bedescribed with reference to FIG. 7.

The upper side portion of FIG. 7 illustrates a likelihood informationimage E1 displayed in a density according to the likelihood informationcorresponding to the moving image Awhile the lower side portion of FIG.7 illustrates a likelihood information image E2 indicating a state inwhich the unnecessary region is being deleted using the user interfaceD2.

For example, operation conventionally performed in units of pixel on aregion in the vicinity of the clipping target object such as a user,specifically, operation such as removal of surrounding dust data,blurring, bordering using touch or pen input is not only delicate workbut also involves difficulty in input and possible erroneous operationsince the target object is moving because it is in a moving image.

In contrast, the information processing apparatus 13 adds, to theuser-clipped moving image B, likelihood information indicating thepossibility that the object is really a target object (the higher thepossibility, the higher the likelihood of the region being an actualtarget object) for each of the pixels of the user-clipped moving imageB, and transmits together. This enables the image processing unit 43 toautomatically adjust the impact degree of the operation in accordancewith the target object likelihood of the pixel of the user-clippedmoving image B.

For example, when calculating the distance to the object, the distancemeasurement unit 22 obtains probability information indicating theprobability of the calculation result being correct and supplies theprobability information to the clipped image generation unit 23. Then,when generating the user-clipped moving image B from the moving image A,the clipped image generation unit 23 adds likelihood informationindicating that the object is a clipping target object in units ofpixel.

The likelihood information image E1 in FIG. 7 indicates that the darkerthe pixel, the higher the likelihood of the object being the targetobject. Then, when the user deletes an unnecessary region in thevicinity of the user using the user interface D2, by performing deletionto allow the impact degree to become higher in inverse proportion to thelikelihood so as not to remove pixels having a certain level oflikelihood or higher, it is possible to make it easy to leave solely thetarget object without carefully tracing the surrounding portion.

In this manner, regarding the deleted unnecessary region, deletion ofthe unnecessary region appearing in the same area is continuouslyperformed even in the following frames.

For example, processing of continuously deleting an unnecessary regionwill be described with reference to FIG. 8.

As illustrated in the clipped moving image E3 in the upper side portionof FIG. 8, there are unnecessary regions in the vicinity of the user andportions close to an end portion of the screen, indicated by grayhatching, that remain unintentionally in the user-clipped moving image Bas a result of being clipped even though it is not a clipping targetobject intended by the user. In order to delete such unnecessaryregions, the user performs operation of deleting the unnecessary regionin the clipped moving image E3 using the eraser-shaped user interface D2as illustrated in FIG. 6.

For example, the clipped moving image E3 includes deletion regions D2-aand D2-b in which the user has performed unnecessary region deletionoperation using the user interface D2.

Then, in the clipped moving image E4 in the frame following the clippedmoving image E3, the position displayed is moved due to movement of theuser. At this time, the unnecessary region might move together with theuser as the target of clipping depending on the characteristic of theunnecessary region, and thus, the unnecessary region might have movedfrom the position designated in the clipped moving image E3 in aprevious frame.

Accordingly, the deletion region D2-b is set to move corresponding tothe movement of the user, with the center of the user's face as areference, for example. In this manner, by recording merely the relativeposition and the relative size from a portion of the range (for example,user's face) automatically detectable among the clipped moving image E3,it is possible to automatically remove an unnecessary region of an aimedsize at an aimed position, in each of the frames. That is, even in acase where the unnecessary region is a region that changes its positionframe by frame, the user can easily continue deletion of suchunnecessary region without designating the position to be deleted foreach of the frames.

Note that this center extraction method is considered to detect a pointthat facilitates feature point analysis of the target object region, forexample, the center position of the face and its size. With thistechnique, with a single instruction given by the user, it is possibleto continuously reflect operation in consecutive moving images even ifthe operation cannot be expressed by affine transformation.

In contrast, since the deletion region D2-a appears fixedly in thevicinity of the end portion of the screen without movement, and thus isfixedly set with the end portion of the screen as a reference, forexample.

FIG. 9 illustrates an example in which the user of the informationprocessing apparatus 13-1 sets a border design around the user-clippedmoving image B1.

For example, the user touches and designates the user-clipped movingimage B1 displayed on the display unit 25, and thereafter operates abutton (not illustrated) for designating border design setting. Inresponse to this, as illustrated in FIG. 9, a combined moving image C3is displayed in which a border design D3 is superimposed so as toenclose the user-clipped moving image B1. Note that the border design D3is displayed not only on the display unit 25 of the informationprocessing apparatus 13-1 but also on the display units 25 of theinformation processing apparatuses 13-2 and 13-3.

Moreover, the user can designate, for example, the type of border to bedesigned (shape, color, and the like), the border region (width from theperson toward the outside), or the degree of blurring of the clippededge. Then, if these items are designated, designated content is addedas operation content to the user-clipped moving image B1, transmitted tothe distribution server 14, and recorded in the operation recording unit42-1. Then, the image processing unit 43-1 can perform image processingof generating the border design D3 in response to user' s movement inaccordance with the operation content recorded in the operationrecording unit 42-1. In this manner, the designated content is reflectedon the border design D3 in real time.

FIG. 10 illustrates an example in which the user of the informationprocessing apparatus 13-1 performs operation of trimming theuser-clipped moving image B1.

For example, the user touches and designates the user-clipped movingimage B1 displayed on the display unit 25, and thereafter operates abutton (not illustrated) for designating trimming operation. In responseto this, as illustrated in FIG. 10, the operation unit 26 displays acombined moving image C4 on which a user interface D4 for performingtrimming is superimposed, on the display unit 25 so as to enclose theuser-clipped moving image B1 in a rectangle.

Then, the user can perform operation of designating a region outside theuser interface D4 that is not partially displayed, by moving a griddisplayed at each of centers of four sides of the user interface D4inward while touching the grid. In accordance with this operation, forexample, operation content (for example, position and size of the userinterface D4) is added to the user-clipped moving image B1 frame byframe and transmitted from the information processing apparatus 13-1.Subsequently, in the distribution server 14, the image processing unit43-1 can perform image processing according to the operation content onthe user-clipped moving image B1. Accordingly, the informationprocessing apparatus 13-1 makes it possible to display the combinedmoving image C4 in which the trimming using the user interface D4 hasbeen reflected on the user-clipped moving image B1 in a state where theuser is moving.

FIG. 11 illustrates an example in which the user of the informationprocessing apparatus 13-1 performs operation of designating depth (Zaxis) coordinates of the user-clipped moving image B1.

For example, the user touches and designates the user-clipped movingimage B1 displayed on the display unit 25, and thereafter operates abutton (not illustrated) for designating the depth coordinates. Inresponse to this, as illustrated in FIG. 11, the operation unit 26displays, on the display unit 25, a combined moving image C5 on whichuser interfaces D5-a and D5-b used for designating the display positionof the user-clipped moving image B1 in the depth direction issuperimposed.

The user interface D5-a is displayed for specifying the object selectedas the target for designating the depth coordinates. In the combinedmoving image C5 illustrated in FIG. 11, the user interface D5-a isdisplayed so as to be superimposed on the user-clipped moving image B,indicating that the user-clipped moving image B is selected as a targetfor designating the depth coordinates. Note that, as the target fordesignating the depth coordinates, for example, it is also possible toselect a virtual object F1 or F2 as illustrated in FIG. 12 to bedescribed below.

The user interface D5-b is an indicator for changing the depthcoordinates, and it is possible to change the depth coordinates of thetarget selected on the user interface D5-a by performing operation ofmoving a slider. For example, setting is designated as a maximum depth(layer on most backward side) by moving the slider of the user interfaceD5-b to the uppermost end, and designated as a minimum depth (layer onmost forward side) by moving the slider of the user interface D5-b tothe lowermost end.

In accordance with this operation performed on the user interface D5-b,for example, operation content (for example, depth coordinates) is addedto the user-clipped moving image B1 frame by frame and transmitted fromthe information processing apparatus 13-1. Then, in the distributionserver 14, the image processing unit 43-1 can set a layer for theuser-clipped moving image B1 in accordance with the operation contentand combining processing by the overall image combining unit 35 isperformed in a sequence corresponding to the layer. Accordingly, theuser of the information processing apparatus 13-1 can set the depthcoordinates of the user-clipped moving image B1 while confirming therelative depth with respect to the user-clipped moving image B2 or B3.

The user interface D5-b will be further described with reference to FIG.12.

A of FIG. 12 illustrates a combined moving image C6 obtained bycombining the user-clipped moving images B4 and B5 with the virtualobjects F1 and F2, and B of FIG. 12 illustrates a layer structure of thecombined moving image C6.

As illustrated in B of FIG. 12, the combined moving image C6 is formedby stacking five layers L1 to L5. A background image is set in the layerL1 and the user-clipped moving image B4 is set in the layer L2, in orderfrom the backward side of the layer structure. Furthermore, the virtualobject F1 (sofa) is set in the layer L3, the user-clipped moving imageB5 is set in the layer L4, and the virtual object F2 (foliage plant) isset in the layer L5.

As a result, as illustrated in A of FIG. 12, in the combined movingimage C6, for example, the user-clipped moving image B4 is displayedbehind the virtual object F1, while the user-clipped moving image B5 isdisplayed in front of the virtual object F1. In addition, the userinterface D5-b displayed in the combined moving image C5 in FIG. 11corresponds to the depth coordinates of the layers L1 to L5 asillustrated in B of FIG. 12.

Meanwhile, in the communication system 11, the background moving imageand the user-clipped moving image B are arranged in a space representedby the X axis direction, the Y axis direction, and the Z axis direction,and the combined moving image C obtained by combining individual imagesis displayed in the display unit 25. In addition, since the combinedmoving image C is displayed two-dimensionally on the display unit 25, itwould be difficult to intuitively perform operation in a depth direction(Z-axis direction) while the user can intuitively perform operationeasily in the two-dimensional directions (X-axis direction and Y-axisdirection).

Accordingly, in a case where the operation of setting depth coordinatesis selected, the communication system 11 can temporarily display alayered structure as illustrated in B of FIG. 12 during the time whenthe operation is being performed. For example, a layered structure inwhich all the moving images used when the overall image combining unit35 generates the combined moving image C are arranged obliquely inaccordance with layer information thereof is transmitted from thedistribution server 14 to the information processing apparatus 13.

This enables the user to visually grasp the depth coordinates of all themoving images and to easily perform operation of setting the desireddepth information.

For example, in a case where the user of the user-clipped moving imageB4 is to be placed in front of the user-clipped moving image B5 andbehind the virtual object F2, it is possible to display a layerstructure as illustrated in B of FIG. 12 and to select a desiredposition (for example, position indicated by a hollow arrow) on the userinterface D5-b. With this operation, it is possible to provide moreintuitive operation than operation of inputting the depth (Z axis)coordinates with an absolute value, for example.

As described above, the user can perform operation on the user-clippedmoving image B using various user interfaces, and can easily reflect theoperation on the combined moving image C.

Meanwhile, assuming that the user-clipped moving image B before theoperation is in a space of three dimensions, that is, the X-axisdirection (horizontal direction), the Y-axis direction (verticaldirection), and the Z-axis direction (depth direction), the operationcontent is classified into one that can express the user-clipped movingimage B after the operation by affine transformation and one that cannotexpress the user-clipped moving image B after the operation by affinetransformation. Note that in the present specification, affinetransformation will be described as an example of image processing thatuses a transformation parameter that is uniquely determinedcorresponding to the user-clipped moving image even if the user moves inimage processing according to the operation content. Alternatively, itis of course possible to apply, in image processing using suchtransformation parameters, transformation other than affinetransformation (for example, projective transformation, similaritytransformation, or inversion transformation).

That is, the operation content that can be expressed by affinetransformation is operation (refer to FIG. 5) such as enlargement orreduction, translation, and rotation of the user-clipped moving image B.For example, when the center coordinates of the user-clipped movingimage B before the operation are fixed as the origin of thetransformation, the operation once designated by the user can becontinuously reflected even in the case of continuous moving images bycontinuing execution of same processing for the target region for eachof the frames.

In contrast, operation content that cannot be expressed by affinetransformation corresponds to operation of trimming a region designatedby the user (refer to FIG. 10), operation of removing a portiondesignated by the user by the pixel flattening processing (refer to FIG.6), or the like, among the user-clipped moving image B.

Accordingly, the operation content transmitted from the informationprocessing apparatus 13 to the distribution server 14 can includeinformation indicating whether the operation can be expressed by affinetransformation, in addition to the information indicating the operationcontent itself.

FIG. 13 is a diagram illustrating the format of the operation contenttransmitted from the information processing apparatus 13 to thedistribution server 14.

As illustrated in the figure, the operation content includes anoperation ID (identification), a target user ID, type information,reference information, a transform flag, remaining time information, anda transformation parameter.

In order to identify all operation content transmitted and received bythe communication system 11, for example, a number that is advanced andregistered in accordance with the order of transmission and reception isset as the operation ID.

The number allocated for identifying the information processingapparatus 13 connected to the communication system 11 is set as thetarget user ID. For example, the target user ID “0” is allocated to theinformation processing apparatus 13-1, the target user ID “1” isallocated to the information processing apparatus 13-1, and the targetuser ID “2” is allocated to the information processing apparatus 13-1.

The information indicating the operation content performed by the useris set as the type information. For example, as described above withreference to FIG. 6, in a case where the user performs operation ofdeleting an unnecessary region, the type information “delete” is set.Moreover, in a case where the user performs operation of enlarging theuser-clipped moving image B using the user interface D1 illustrated inFIG. 5, the type information “enlargement” is set. Moreover, in a casewhere the user performs operation of achieving a presentation effect bya predetermined animation, for example, the type information “animation”is set. Furthermore, in a case where the user performs operation oftrimming the user-clipped moving image B using the user interface D4illustrated in FIG. 10, the type information “trimming” is set.

Information indicating a position to be a reference in reflecting theoperation content is set as reference position information. For example,as described above with reference to FIG. 8, reference information “facecenter” is set in a case where operation of deleting an unnecessaryregion with reference to the center of the face of the user-clippedmoving image B is performed. Similarly, as described above withreference to FIG. 8, in a case where operation of deleting anunnecessary region around the upper right end of the user interface D2-ais performed, the reference information “upper right” is set.

Information indicating whether the operation content is reflected byaffine transformation is set in the transform flag. For example, in acase where the operation content is deletion or trimming, a transformflag “N” indicating that the operation content is not reflected byaffine transformation is set. In contrast, in a case where the operationcontent is enlargement or animation, a transform flag “Y” indicatingthat the operation content is reflected by transformation is set.

Information indicating remaining time until the reflection of theoperation content is finished is set in the remaining time information.For example, in a case where the time for reflecting the operationcontent is not designated when the operation is performed, remainingtime information “infinity” is set. Moreover, in a case where the timefor reflecting the operation content is designated when the operation isperformed, for example, in a case where the time is 25 seconds, theremaining time information is set as “t=25”.

Various parameters needed for reflecting the operation content are setas the transformation parameter. For example, as described above withreference to FIG. 8, in a case where a user performs operation ofdeleting an unnecessary region, a transformation parameter of“positional relationship with respect to a relative center andintensity” indicating a positional relationship with respect to aposition set as a deletion reference and intensity in performingdeletion is set. Moreover, in a case where the user performs operationof enlarging the user-clipped moving image Busing the user interface D1illustrated in FIG. 5, a transformation parameter of “enlargement centerand enlargement rate” indicating a center position set as an enlargementreference and an enlargement rate in performing enlargement is set.

Moreover, in a case where the user performs operation of achieving apresentation effect by a predetermined animation, for example,“animation ID” designating animation is set as the transformationparameter. Moreover, in a case where the user performs operation oftrimming the user-clipped moving image B using the user interface D4illustrated in FIG. 10, a transformation parameter “relative positionfrom the center, relative size” indicating the position for performingrelative trimming with respect to the position set as the trimmingreference and the size is set.

With transmission of the operation content in the format as describedabove from the information processing apparatus 13 to the distributionserver 14, it is possible on the distribution server 14 to perform imageprocessing according to the operation content.

Next, FIG. 14 is a flowchart for describing processing performed on theinformation processing apparatus 13 in FIG. 3.

For example, the processing is started after the user performs operationon the operation unit 26 to prompt participation in a service providedin the communication system 11. In step S11, the digital signalprocessing unit 21 obtains a moving image A (refer to FIG. 2) capturedby an imaging device (not illustrated) and applies digital signalprocessing to the moving image A, and supplies image data for each ofthe frames constituting the moving image A to the clipped imagegeneration unit 23.

In step S12, the distance measurement unit 22 calculates, for example, aparallax between a plurality of moving images, generates a depth mapthat maps a distance from the imaging device to the object for each ofthe pixels, and supplies the depth map to the clipped image generationunit 23.

In step S13, the clipped image generation unit 23 performs imageprocessing of clipping a target object from the moving image A suppliedfrom the digital signal processing unit 21 in step S11 on the basis ofthe depth map supplied from the distance measurement unit 22 in stepS12. With this processing, the clipped image generation unit 23generates the user-clipped moving image B and supplies the image to thecommunication unit 24.

In step S14, the communication unit 24 determines whether there has beenoperation performed by the user. For example, if the operation contentcorresponding to the operation by the user is supplied from theoperation unit 26, the communication unit 24 determines that there hasbeen operation by the user. If the operation content is not suppliedfrom the operation unit 26, the communication unit 24 determines thatthere has been no operation by the user.

In a case where it is determined instep S14 that there has beenoperation by the user, the processing proceeds to step S15, thecommunication unit 24 adds the operation content to the user-clippedmoving image B supplied from the clipped image generation unit 23 instep S13 and transmits the image to the distribution server 14.

In contrast, in a case where it is determined in step S14 that there hasbeen no operation by the user, the processing proceeds to step S16, andthe communication unit 24 transmits the user-clipped moving image Bsupplied from the clipped image generation unit 23 in step S13 to thedistribution server 14.

After the processing of step S15 or S16, the processing proceeds to stepS17, and the communication unit 24 receives the combined moving image Ctransmitted from the distribution server 14 and supplies the image tothe display unit 25, and then, the display unit 25 displays the combinedmoving image C. Thereafter, the processing returns to step S11 andsimilar processing is then repeated.

As described above, the information processing apparatus 13 can transmitthe user-clipped moving image B obtained by clipping the user from themoving image A to the distribution server 14. Moreover, in a case wherethe user performs operation, the operation content can be added to theuser-clipped moving image B and transmitted to the distribution server14.

Next, FIG. 15 is a flowchart for describing the processing performed inthe distribution server 14 in FIG. 4.

For example, processing is started when there is notification from aplurality of information processing apparatuses 13 that they wouldparticipate in a service provided in the communication system 11. Instep S21, the communication unit 31 receives the user-clipped movingimage B transmitted from the information processing apparatus 13 andsupplies the received user-clipped moving image B to the user managementunit 32.

In step S22, the user management unit 32 refers to the target user IDadded to the user-clipped moving image B, for example, and transmits theuser-clipped moving image B to the user individual processing unit 33corresponding to the information processing apparatus 13 that hastransmitted the user-clipped moving image B. With this processing, theuser individual processing unit 33 records the user-clipped moving imageB in the individual image recording unit 41, and if the operationcontent has been added to the user-clipped moving image B, the operationcontent is recorded in the operation recording unit 42.

In step S23, the image processing unit 43 individually performs imageprocessing on the user-clipped moving image B in the user individualprocessing unit 33 in which processing content of non-affinetransformation is recorded in the operation recording unit 42, among thepredetermined number of user individual processing units 33corresponding to the number of information processing apparatuses 13connected to the communication system 11. That is, the image processingaccording to the operation content in which the transform flag “N” isset in the operation content recorded in the operation recording unit 42of the user individual processing unit 33, for example, image processingcorresponding to operation such as trimming and unnecessary regionremoval is individually performed.

In step S24, the overall image combining unit 35 requests theuser-clipped moving image B in descending order of depth from thepredetermined number of user individual processing units 33.

In step S25, the predetermined number of user individual processingunits 33 determine whether the user-clipped moving image B has beenrequested. Then, on the user individual processing unit 33 thatdetermines that the user-clipped moving image B has not been requested,the processing returns to step S24 and the processing is suspended. Incontrast, the user individual processing unit 33 that determines thatthe user-clipped moving image B has been requested becomes a target towhich the user-clipped moving image B is transmitted, and the processingproceeds to step S26.

In step S26, the target user individual processing unit 33 supplies theuser-clipped moving image B in response to the request from the overallimage combining unit 35. At this time, if the operation content forwhich the transform flag “Y” is set is recorded in the operationrecording unit 42, the target user individual processing unit 33 alsosupplies the processing content together with the user-clipped movingimage B to the overall image combining unit 35.

Then, the overall image combining unit 35 superimposes the user-clippedmoving image B on the background moving image supplied from thebackground recording unit 34 in descending order of depth. At this time,if the processing content of affine transformation has been supplied,the overall image combining unit 35 performs image processing reflectingthe processing content, for example, image processing corresponding tooperation such as enlargement and reduction, and superimposes theuser-clipped moving image B on the combined moving image C.

In step S27, the overall image combining unit 35 determines whether allthe user-clipped moving images B corresponding to the predeterminednumber of user individual processing units 33 have been combined.

In a case where the overall image combining unit 35 determines in stepS27 that all the user-clipped moving images B corresponding to thepredetermined number of user individual processing units 33 have notbeen combined, the processing returns to step S24. Then, similarprocessing is repeated for the user-clipped moving image B having thenext greatest depth as a target.

In contrast, in step S27, in a case where the overall image combiningunit 35 determines that all the user-clipped moving images Bcorresponding to the predetermined number of user individual processingunits 33 have been combined, the processing proceeds to step S28.

In step S28, the overall image combining unit 35 supplies the generatedcombined moving image C to the communication unit 31, and transmits thecombined moving image C to the information processing apparatus 13 viathe network 12. Thereafter, the processing returns to step S21, andsimilar processing is repeated for the next frame as a processingtarget.

As described above, the distribution server 14 can perform imageprocessing for non-affine transformation processing individually and canperform image processing for affine transformation processing whencombining the user-clipped moving image B. Additionally, it is possibleto generate the combined moving image C by combining all theuser-clipped moving images B in descending order of depth, and totransmit the combined moving image C to the information processingapparatus 13.

Note that the method by which the distance measurement unit 22 of theinformation processing apparatus 13 calculates the distance to theobject is not limited to the above-described method of calculating theparallax, but may include various methods capable of generating a mapfor clipping the target object. For example, the distance measurementunit 22 may adopt a method of calculating the distance on the basis of atime difference from a point of emitting an infrared ray to the objectuntil the infrared ray is reflected and returned, a method in whichpatterned light is emitted and the distance is calculated on the basisof the degree of deformation of the pattern according to the surface ofthe object.

Moreover, for example, it is possible to generate a moving imageincluding a region in which the user is photographed (moving imageexcluding the regions other than the region in which the user isphotographed) by imaging using a specific color such as green back orblue back as a background with a chroma key compositing technique andeliminating the background by making the component of the specific colortransparent. That is, it is possible to generate a moving imageincluding the region in which the use is photographed using a method ofclipping the region in which the user is photographed to generate auser-clipped moving image, or other methods, and to combine the movingimage into a combined moving image.

Moreover, sharing of functions provided as the whole communicationsystem 11 including the information processing apparatus 13 and thedistribution server 14, by the information processing apparatus 13 andthe distribution server 14, is not limited to the above-describedconfiguration illustrated in FIGS. 3 and 4. For example, the imageprocessing for each of the user-clipped moving images B performed by theuser individual processing unit 33 in the distribution server 14 may beperformed in the information processing apparatus 13, and theuser-clipped moving image B that has undergone image processing on theinformation processing apparatus 13 may be transmitted.

Moreover, there is no need to process the function of the informationprocessing apparatus 13 within one apparatus. For example, thesefunctions may be implemented as software on an information processingterminal such as a personal computer or a smartphone, digital signalsmay be obtained from an external camera device, and subsequentprocessing may be performed.

FIG. 16 is a block diagram illustrating a modification example of theinformation processing apparatus 13.

As illustrated in FIG. 16, the information processing apparatus 13Aincludes two individual apparatuses, that is, an image transmissionapparatus 51 and a user interface apparatus 52.

The image transmission apparatus 51 includes a digital signal processingunit 21, a distance measurement unit 22, a clipped image generation unit23, and a communication unit 24-1, and transmits the user-clipped movingimage B generated by the clipped image generation unit 23 to thedistribution server 14. For example, the image transmission apparatus 51can be provided by software as a portion of functions of a camera devicehaving an imaging function.

The user interface apparatus 52 includes a communication unit 24-2, adisplay unit 25, and an operation unit 26, displays the combined movingimage C distributed from the distribution server 14 on the display unit25, and transmits the operation content input by the user's operation onthe operation unit 26 to the distribution server 14. For example, theuser interface apparatus 52 can employ a web browser that provides inputof operation content and display of images by software. In this case,the image displayed on the user interface apparatus 52, the operationcontent that has been input, and the like, are held as a web program onthe side of the distribution server 14, and the user can use thefunction of the distribution server 14 by accessing the distributionserver 14 from the web browser.

Furthermore, the communication system 11 may be configured to allow thedisplay to change in response to the user's operation on the operationunit 26 as described above, or to change by interaction between theuser-clipped moving images B, for example.

A change in display by interaction between user-clipped moving images Bwill be described with reference to FIG. 17.

As illustrated in a combined moving image C7-1 in the upper portion ofFIG. 17, the user-clipped moving images B1 and B2 are displayed side byside. Moreover, layers of the user-clipped moving images B1 and B2 areset to the same depth coordinates.

At this time, as illustrated in a combined moving image C7-2 in themiddle portion of FIG. 17, it is assumed that the user of theinformation processing apparatus 13-2 is inclined toward the user of theinformation processing apparatus 13-1 at a certain level ofacceleration, and the head of the user-clipped moving image B2 touchesthe head of the user-clipped moving image B1. In this case, the overallimage combining unit 35 can interpret the interaction as, for example,head-butting operation, and performs image processing according to theinteraction.

With this processing, as illustrated in a combined moving image C7-3 inthe lower portion of FIG. 17, the overall image combining unit 35 canperform image processing such that the user-clipped moving image B1jumps while spinning.

The content of a display change corresponding to the interaction betweenthe user-clipped moving images B and a condition for producing thedisplay change can be set in advance by an administrator who providesthe service by the communication system 11. Moreover, individual usersmay set the content and condition exclusively for themselves.

In this manner, in the communication system 11, it is possible to changethe display automatically (without operation) corresponding to thepredetermined interaction on the basis of the positional relationshipand situation between the user-clipped moving images B in addition tooperation using the user interface. Moreover, for example, it ispossible to change the display automatically (without operation) inaccordance with the positional relationship and the situation betweenthe user-clipped moving image B and the background moving image.

Next, another exemplary processing performed by the communication system11 will be described with reference to FIGS. 18 and 19.

In FIG. 18, the processing of reflecting preregistered operation contentat a designated time will be described.

As described above, in the communication system 11, various types ofoperation content are reflected on the user-clipped moving image in realtime, and a combined moving image that has undergone image processingaccording to the operation content is distributed. On the other hand,the communication system 11 can be configured, for example, to allow theoperation content to be preregistered so as to be reflected at adesignated time, and to allow a combined moving image that has undergonethe image processing according to the operation content to bedistributed at a timing when the designated time arrives, instead ofreflecting the operation content on the target object in real time.

An operation content registration screen 101 illustrated in FIG. 18 isused for preregistering the operation content, and displays a previewwindow 102, a work window 103, a tool bar 104, and a time managementwindow 105.

The preview window 102 displays a current frame of the combined movingimage distributed from the distribution server 14 and a layer structureof the frame. The example in FIG. 18 displays three layers and a layernumber indicating the order of the individual layer at the upper left ofthe frame representing each of the layers. For example, the previewwindow 102 displays a combined moving image in which a background movingimage is set in a layer with layer number 1, a person A on the left sideis set in a layer with layer number 2, and a person B on the right sideis set in a layer with layer number 3.

The work window 103 displays a layer for designating operation content.The example in FIG. 18 illustrates a state in which operation isperformed on five layers, and displays a layer number indicating theorder of the individual layer at the upper left of the framerepresenting each of the layers. For example, similarly to the previewwindow 102, letters A arranged on the left side are set in a layer withlayer number 4 and letters B arranged on the right side are set in alayer with layer number 5, in addition to the layers with layer numbers1 to 3.

The tool bar 104 displays buttons for selecting various tools to be usedwhen designating the operation content.

The time management window 105 displays a time bar for each of thelayers displayed in the work window 103. The time bar is provided todesignate the timing of reflecting the operation content registered inthe work window 103. For example, an arrow displayed at the left end ofthe time bar indicates the time of the currently displayed frame, andthe operation content designated using the time bar is displayed so asto flow toward the left side with the lapse of time.

On this operation content registration screen 101, for example, it ispossible to register in advance the timing of starting the reflection ofthe operation content and the timing of completing the reflection of theoperation content, in association with the operation content.

For example, the operation content registration screen 101 illustratesan exemplary case where the operation content in which the displaymagnification of the combined moving image is currently set to 1 time(×1.0) and zooming is started from a predetermined timing to set thedisplay magnification of the combined moving image to 1.25 times hasbeen registered. For example, the display magnification change isstarted from the timing of the arrow illustrated on a leftward portionin the vicinity of the center on the operation content registrationscreen 101, and the display magnification is changed to 1.25 times(×1.25) at the timing of the arrow illustrated on the right thereof, atwhich the two people and the background are enlarged and displayed.

Moreover, on the operation content registration screen 101, operationcontent of displaying the letters A and B are registered during theperiod in which the display magnification is being changed. Accordingly,the letters A and B are displayed at the timing when zooming is started,and the letters A and B turn into non-display states at the timing whenzooming is finished. At this time, the letters A and B are also enlargedand displayed in accordance with the change of the displaymagnification.

In this manner, by transmitting the operation content preregisteredusing the operation content registration screen 101 from the informationprocessing apparatus 13 to the distribution server 14 in accordance withthe timing of reflecting the operation content, it is possible in thecommunication system 11 to distribute the combined moving image that hasundergone the image processing according to the operation content at atiming when the designated time arrives.

Furthermore, the communication system 11 is capable of determining theoperation content to be reflected in accordance with a predeterminedmethod in a case where a plurality of users has preregistered theoperation content at a same timing.

A method of determining the operation content to be reflected will bedescribed with reference to FIG. 19.

An operation content determination screen 111 illustrated in FIG. 19 isused in a method of determining operation content to be reflected byviewer's voting for operation content preregistered by a large number ofusers. As illustrated in the figure, the operation content determinationscreen 111 includes a preview window 112, a trial view window 113, andan entry window 114.

The preview window 112 displays a current frame of the combined movingimage distributed from the distribution server 14.

The trial view window 113 displays a combined moving image when acertain user views, on a trial basis, the combined moving imagereflecting the preregistered operation content. Then, in a case ofvoting for the operation content after viewing, on a trial basis, themoving image in which the operation content has been reflected,operation is performed on a good button arranged in the trial viewwindow 113.

The entry window 114 displays operation content preregistered by varioususers using bands for individual time axes. Then, by touching the banddisplayed in the entry window 114, the combined moving image in whichthe operation content has been reflected is displayed in the trial viewwindow 113.

Note that while the operation content determination screen 111illustrated in FIG. 19 describes an exemplary case of collectivelyvoting operation on all layers of a combined moving image, it is alsopossible to vote for each of the layers, such as a background layer anda person layer. For example, the voting method can be selected when adistributor of a moving image starts distribution. In this case, theentry window 114 displays, for each of layers, bands of operationcontent preregistered for each of the layers. Then, voting is performedfor each of the layers, and in a case where the most voted items are theoperation content preregistered by different users between thebackground layer and the person layer, for example, the most voted itemsare employed in combination with each other.

Note that the method of determining the preregistered operation contentis not limited to the method of determining by this type of voting. Forexample, in a case where materials that can be selected as options(heart, sparkle, musical notes, etc.) are used, the operation contentmay be determined by a method adopting operation content in which thosematerials are used most frequently. Alternatively, the operation contentmay be determined by selection of a certain user by a method in whichthe user (for example, distributor of the moving image) has theauthority of decision.

Furthermore, while in the above-described present embodiment, processingof combining all the user-clipped moving images transmitted from theinformation processing apparatuses 13-1 to 13-3 in the distributionserver 14 has been described, it is also possible, for example, tocombine a user-clipped moving image transmitted from at least oneinformation processing apparatus 13 with the background moving image anddistribute the combined image. Furthermore, it is also possible tocombine a virtual object generated by computer graphics instead of anobject clipped from the captured moving image, with one user-clippedmoving image.

Note that the processing described with reference to the above-describedflowcharts need not necessarily be processed in chronological orderdescribed as a flowchart, and may include processing executed inparallel or individually (e.g., parallel processing or processing byobject). Moreover, the program may be processed by one CPU, or may beprocessed distributively by a plurality of CPUs.

Moreover, the above-described series of processing (informationprocessing method) can be executed by hardware or by software. In a casewhere the series of processing is executed by software, a programconstituting the software is installed from a program recording mediumstoring the program into a computer incorporated in dedicated hardware,or into a general-purpose computer, for example, capable of executingvarious functions by installing various programs.

FIG. 20 is a block diagram illustrating an exemplary configuration ofhardware of a computer in which the series of processing described aboveis executed by a program.

In the computer, a central processing unit (CPU) 201, a read only memory(ROM) 202, and a random access memory (RAM) 203 are interconnected witheach other via a bus 204.

The bus 204 is further connected with an input/output interface 205. Theinput/output interface 205 is connected with an input unit 206 includinga keyboard, a mouse, a microphone and the like, an output unit 207including a display, a speaker and the like, a storage unit 208including a hard disk, a nonvolatile memory and the like, acommunication unit 209 including a network interface and the like, and adrive 210 configured to drive a removable medium 211 such as a magneticdisk, an optical disk, a magneto-optical disk, and a semiconductormemory.

In the computer configured as above, the series of above-describedprocessing is executed by operation in which the CPU 201 loads, forexample, a program stored in the storage unit 208 onto the RAM 203 viathe input/output interface 205 and the bus 204 and executes the program.

The program executed by the computer (CPU 201) is, for example, providedas a program recorded on the removable medium 211 which is a packagemedium such as a magnetic disk (including a flexible disk), an opticaldisk (including a compact disc-read only memory (CD-ROM) and a digitalversatile disc (DVD)), a magneto-optical disk, or a semiconductormemory, or provided via a wired or wireless transmission medium such asa local area network, the Internet, and digital satellite broadcast.

Then, the program can be installed in the storage unit 208 via theinput/output interface 205, by attaching the removable medium 211 to thedrive 210. In this case, the program can be received at thecommunication unit 209 via a wired or wireless transmission medium andbe installed in the storage unit 208. Alternatively, the program can beinstalled in the ROM 202 or the storage unit 208 beforehand.

Note that the present technology may also be configured as follows.

-   (1)

An image processing system including:

an image processing unit configured to perform, individually for each ofobjects, image processing on an object moving image generated from amoving image in which a predetermined object is imaged and generated soas to include a region in which the object is photographed, the imageprocessing corresponding to operation content applied to the objectmoving image; and

a combined image generation processing unit configured to generate acombined moving image including at least one of the objects by combiningthe object moving image that has undergone image processing with anothermoving image.

-   (2)

The image processing system according to (1), further including:

an individual image recording unit configured to record the objectmoving image for each of the plurality of objects; and

an operation recording unit configured to record the operation contentfor each of the plurality of objects every time operation is performedon the object moving image,

in which the image processing unit reflects latest operation contentrecorded in the operation recording unit on the object moving imagerecorded in the individual image recording unit.

-   (3)

The image processing system according to (1) or (2), further including:

a distance measurement unit configured to measure a distance to theobject photographed in the moving image; and

a clipped image processing unit configured to perform image processingof clipping a predetermined object from the moving image in which thepredetermined object is imaged on the basis of the distance to theobject measured by the distance measurement unit.

-   (4)

The image processing system according to (3),

in which the distance measurement unit adds likelihood informationindicating probability of a calculation result being correct to theobject moving image in units of pixels constituting the moving imagewhen calculating the distance to the object, and

the image processing unit performs image processing of deleting anunnecessary region with reference to the likelihood information in acase where operation of deleting the unnecessary region is performed ona state in which the unnecessary region to be naturally deleted by theclipped image processing unit remains.

-   (5)

The image processing system according to (4),

in which the image processing unit performs image processing of deletingthe unnecessary region in accordance with a relative position and arelative size from one region portion as a center that is automaticallydetectable among the object moving image.

-   (6)

The image processing system according to (4) or (5),

in which the image processing unit performs image processing of deletingthe unnecessary region fixedly appearing in the moving image to be atarget of generating the object moving image, in accordance with aposition set on the basis of the moving image.

-   (7)

The image processing system according to any of (1) to (6),

in which the image processing unit performs image processing of trimmingthe object moving image on the basis of a position at which the relativetrimming is performed relative to a reference position set as areference of the trimming as a center and size thereof, in accordancewith the operation content of performing trimming on the object movingimage.

-   (8)

The image processing system according to any of (1) to (7),

in which the image processing unit performs image processingcorresponding to the operation content applied to the object movingimage, using a transformation parameter uniquely determinedcorresponding to the object moving image even in a case where the objectmoves.

-   (9)

The image processing system according to (8), in which the imageprocessing unit performs image processing using the transformationparameter in accordance with the operation content of performingenlargement, reduction, translation, or rotation, on the object movingimage.

-   (10)

The image processing system according to any of (1) to (9),

in which preregistration is performed such that the operation content isreflected at a designated time, and

the image processing unit performs image processing in accordance withthe registered operation content on the object moving image at a timingwhen the designated time arrives.

-   (11)

The image processing system according to (10),

in which, in a case where a plurality of items of the operation contentis preregistered in advance so as to be reflected at a predetermineddesignated time, voting is performed on operation content to be actuallyreflected among the plurality of items of operation content, and

the image processing unit performs image processing corresponding to themost voted operation content on the object moving image at the timingwhen the designated time arrives.

-   (12)

An image processing method including the steps of:

performing, individually for each of objects, image processing on anobject moving image generated from a moving image in which apredetermined object is imaged and generated so as to include a regionin which the object is photographed, the image processing correspondingto operation content applied to the object moving image; and

generating a combined moving image including at least one of the objectsby combining the object moving image that has undergone image processingwith another moving image.

-   (13)

A program for causing a computer to execute image processing includingthe steps of:

performing, individually for each of objects, image processing on anobject moving image generated from a moving image in which apredetermined object is imaged and generated so as to include a regionin which the object is photographed, the image processing correspondingto operation content applied to the object moving image; and

generating a combined moving image including at least one of the objectsby combining the object moving image that has undergone image processingwith another moving image.

Note that the present embodiment is not limited to the above-describedembodiments but can be modified in a variety of ways within a scope ofthe present disclosure.

REFERENCE SIGNS LIST

-   11 Communication system-   12 Network-   13-1 to 13-3 Information processing apparatus-   14 Distribution server-   21 Digital signal processing unit-   22 Distance measurement unit-   23 Clipped image generation unit-   24 Communication unit-   25 Display unit-   26 Operation unit-   31 Communication unit-   32 User management unit-   33-1 to 33-3 User individual processing unit-   34 Background recording unit-   35 Overall image combining unit-   41-1 to 41-3 Individual image recording unit-   42-1 to 42-3 Operation recording unit-   43-1 to 43-3 Image processing unit-   51 Image transmission apparatus-   52 User interface apparatus

1. An image processing system comprising: an image processing unitconfigured to perform, individually for each of objects, imageprocessing on an object moving image generated from a moving image inwhich a predetermined object is imaged and generated so as to include aregion in which the object is photographed, the image processingcorresponding to operation content applied to the object moving image;and a combined image generation processing unit configured to generate acombined moving image including at least one of the objects by combiningthe object moving image that has undergone image processing with anothermoving image.
 2. The image processing system according to claim 1,further comprising: an individual image recording unit configured torecord the object moving image for each of the plurality of objects; andan operation recording unit configured to record the operation contentfor each of the plurality of objects every time operation is performedon the object moving image, wherein the image processing unit reflectslatest operation content recorded in the operation recording unit on theobject moving image recorded in the individual image recording unit. 3.The image processing system according to claim 1, further comprising: adistance measurement unit configured to measure a distance to the objectphotographed in the moving image; and a clipped image processing unitconfigured to generate the object moving image by performing imageprocessing of clipping a predetermined object from the moving image inwhich the predetermined object is imaged on the basis of the distance tothe object measured by the distance measurement unit.
 4. The imageprocessing system according to claim 3, wherein the distance measurementunit adds likelihood information indicating probability of a calculationresult being correct to the object moving image in units of pixelsconstituting the moving image when calculating the distance to theobject, and the image processing unit performs image processing ofdeleting an unnecessary region with reference to the likelihoodinformation in a case where operation of deleting the unnecessary regionis performed on a state in which the unnecessary region to be naturallydeleted by the clipped image processing unit remains.
 5. The imageprocessing system according to claim 4, wherein the image processingunit performs image processing of deleting the unnecessary region inaccordance with a relative position and a relative size from one regionportion as a center that is automatically detectable among the objectmoving image.
 6. The image processing system according to claim 4,wherein the image processing unit performs image processing of deletingthe unnecessary region fixedly appearing in the moving image to be atarget of generating the object moving image, in accordance with aposition set on the basis of the moving image.
 7. The image processingsystem according to claim 1, wherein the image processing unit performsimage processing of trimming the object moving image on the basis of aposition at which the relative trimming is performed relative to areference position set as a reference of the trimming as a center andsize thereof, in accordance with the operation content of performingtrimming on the object moving image.
 8. The image processing systemaccording to claim 1, wherein the image processing unit performs imageprocessing corresponding to the operation content applied to the objectmoving image, using a transformation parameter uniquely determinedcorresponding to the object moving image even in a case where the objectmoves.
 9. The image processing system according to claim 8, wherein theimage processing unit performs image processing using the transformationparameter in accordance with the operation content of performingenlargement, reduction, translation, or rotation, on the object movingimage.
 10. The image processing system according to claim 1, whereinpreregistration is performed such that the operation content isreflected at a designated time, and the image processing unit performsimage processing in accordance with the registered operation content onthe object moving image at a timing when the designated time arrives.11. The image processing system according to claim 10, wherein, in acase where a plurality of items of the operation content ispreregistered in advance so as to be reflected at a predetermineddesignated time, voting is performed on operation content to be actuallyreflected among the plurality of items of operation content, and theimage processing unit performs image processing corresponding to themost voted operation content on the object moving image at the timingwhen the designated time arrives.
 12. An image processing methodcomprising the steps of: performing, individually for each of objects,image processing on an object moving image generated from a moving imagein which a predetermined object is imaged and generated so as to includea region in which the object is photographed, the image processingcorresponding to operation content applied to the object moving image;and generating a combined moving image including at least one of theobjects by combining the object moving image that has undergone imageprocessing with another moving image.
 13. A program for causing acomputer to execute image processing comprising the steps of:performing, individually for each of objects, image processing on anobject moving image generated from a moving image in which apredetermined object is imaged and generated so as to include a regionin which the object is photographed, the image processing correspondingto operation content applied to the object moving image; and generatinga combined moving image including at least one of the objects bycombining the object moving image that has undergone image processingwith another moving image.